feat(blaze): prune checkpoint history - #2474
Conversation
5804993 to
88921b2
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 88921b2483
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
88921b2 to
91a138e
Compare
|
@codex review Prune with tightened BadRequest handling for uncommitted protected identifiers. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 91a138e20b
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
91a138e to
06bd020
Compare
|
@codex review Prune now reports post-tombstone cleanup failures as PruneOutcome::PartialCleanup and the manager marks the sandbox RecoveryRequired. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 06bd020fdc
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
06bd020 to
9af5bdd
Compare
|
@codex Review |
|
@codex review Prune now classifies every post-rename uncertainty as PruneOutcome::PartialCleanup and rejects prune on a RecoveryRequired sandbox with HTTP 409. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9af5bdd952
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Add POST /v1/sandboxes/{id}/checkpoints/prune. The endpoint verifies the
committed catalog, protects HEAD together with its full parent chain and
the ancestor chains of any explicitly listed identifiers, and removes only
unreachable entries.
Publish a named tombstone before renaming a candidate directory so a
restart can identify the pruned candidate and finish removing it. The
paired owner file records the sandbox, checkpoint, nonce, and directory
identity, and restart cleanup already treats prune tombstones and owner
records the same way as capture staging leftovers.
Document the route, protection rules, and restart behavior in the English
and Chinese operator README and runtime user guide.
Signed-off-by: Weisson <Weisson@linux.alibaba.com>
9af5bdd to
9fcfb42
Compare
|
@codex review sweep_candidate now classifies every per-candidate outcome (Removed / Retained / RemovedWithRetainedScratch) so a partial success is never reported as a plain HTTP 500, and owner-publication failures clean up scratch or force RecoveryRequired. |
|
Codex Review: Something went wrong. Try again later by commenting “@codex review”. ℹ️ About Codex in GitHubCodex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback". |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9fcfb424fb
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| // Publishing the owner file creates prune scratch. Any failure between | ||
| // creation and the rename must remove that scratch, otherwise a later | ||
| // `Complete` result would contradict its documented invariant. | ||
| let owner = retain!(write_json_new(sandbox, &owner_name, &owner_record)); |
There was a problem hiding this comment.
Recover owner files when publication fails
When writing or syncing the owner fails after its O_EXCL creation—for example, due to ENOSPC or an I/O error—write_json_new returns an error here and retain! classifies it as Retained, so no cleanup path receives the file handle. With no earlier removals, the manager consequently returns HTTP 500 and leaves the sandbox Running while a partial .prune.*.owner file remains and later prune calls ignore it. Fresh evidence after the prior fix is that write_json_new still performs the fallible writes and sync_all internally at lines 1828–1833 before returning the owner handle; expose the created file for cleanup or classify this as recovery-required.
AGENTS.md reference: src/blaze/AGENTS.md:L32-L36
Useful? React with 👍 / 👎.
Why
Blaze can commit and list checkpoints but has no way to reclaim the disk
space held by checkpoints that neither the current HEAD nor any explicitly
protected identifier can still reach. Operators need a bounded, restart-safe
way to remove unreachable checkpoint history without touching the active
capture chain.
What changed
POST /v1/sandboxes/{id}/checkpoints/prune. The request body isoptional; when present it must be a JSON object with only a
protectedarray of canonical
ckpt-<uuid>identifiers. The response reports theremoved count and identifiers.
ancestor chains of every explicitly protected identifier. Unknown protected
identifiers are rejected with HTTP 400 before any deletion; the endpoint
otherwise returns HTTP 200 with the removed list (which may be empty).
owner file records the sandbox, checkpoint, nonce, and directory identity
so restart cleanup can identify each pruned candidate and finish removing
it.
renameat NOREPLACEprevents silently overwriting a concurrentcandidate; identity revalidation between planning and rename rejects a
catalog that has changed since the plan was built.
classify_scratch_nameto recognize.prune.<checkpoint>.<nonce>tombstone directories and their
.ownerfiles.remove_sandboxand theexisting scratch cleanup already act on these categories, so restart
cleanup or destroy removes any interrupted prune leftovers.
lock and run the actual scan on the blocking pool.
restart behavior in the English and Chinese README and runtime user guide.
No
/v1/instancescompatibility route is added. No dependency on #2473,#2475, or #2471.
Related issue
Follow-up on #2472 (capture and list). Delivers the pruning entry point that
was intentionally deferred.
User / Agent impact
Operators and agents can reclaim disk space held by unreachable checkpoint
branches without stopping or destroying the sandbox. The endpoint acquires
the same per-sandbox operation lock as capture and list, so callers do not
observe partial history changes.
Risk and compatibility
This adds one HTTP endpoint and two new scratch-name categories inside the
sandbox checkpoint namespace. Existing catalogs remain compatible: no new
files appear until an operator calls prune, and any interrupted prune leaves
only well-formed tombstone and owner files that the existing catalog
cleanup already handles.
Prune never touches the HEAD-reachable chain, so a caller with the wrong
protectedlist will at worst leave more history than intended; a repeatcall with an updated list is safe. The endpoint fails closed before deletion
on any protected identifier that is not currently committed.
Validation
Exact candidate:
7a8005874e078455e9acde0f4f6981839919489d(main, includes merged feat(blaze): capture sandbox checkpoints #2472)9fcfb424fbdca747aec439a928c2c835547597a5e5f2bc3aa30aa7ee998ebb308d5c0cb7798a9ff9An uninterrupted Linux x86_64 run on Rust 1.88.0 completed for the tree above.
It used a fresh source directory, a dedicated Cargo home, and an initially
empty task-exclusive target directory for each stage. Every Cargo stage ran
with
--locked --offline.cargo fmt --all -- --checkcargo metadata, default and all featurescargo build --workspace --all-targets, default and all featurescargo clippy --workspace --all-targets -- -D warnings, default and all featurescargo test --workspace -- --test-threads=1cargo test --workspace --all-features -- --test-threads=1RUSTDOCFLAGS=-D warnings cargo doc --workspace --no-deps, default and all featuresscripts/docs-lint.shscripts/docs-link-check.pygit diff --checkFocused prune coverage:
checkpoint_store::tests::prune_preserves_head_and_explicit_lineagesprovesthe HEAD chain and every ancestor of a protected identifier survive while an
unreachable branch is removed.
checkpoint_store::tests::prune_rejects_a_protected_id_that_is_not_committedproves an unknown protected identifier is rejected before deletion.
checkpoint_store::tests::prune_leaves_a_missing_sandbox_directory_untouchedproves a sandbox without any checkpoint namespace remains untouched.
checkpoint_store::tests::prune_removes_owner_and_tombstone_after_completionproves the sandbox directory contains no prune scratch after a successful
prune.
api::tests::prune_route_reports_no_removals_when_history_is_all_reachableproves the HTTP contract for a fully-reachable history.
api::tests::prune_route_rejects_unknown_body_fieldsproves the requestbody accepts only
protected.api::tests::prune_route_rejects_unknown_protected_checkpoint_as_bad_requestproves a canonical but uncommitted protected identifier answers HTTP 400.
checkpoint_store::tests::prune_reports_partial_cleanup_when_post_tombstone_work_failsproves a failure after the tombstone rename returns
PruneOutcome::PartialCleanupwith the confirmed removed prefix, retains the prune scratch for later
cleanup, and leaves the HEAD chain intact.
api::tests::prune_route_rejects_a_recovery_required_sandboxproves arecovery-required sandbox answers HTTP 409 instead of silently accepting a
prune retry that would bypass retained scratch.
Documentation and rollback
The English and Chinese Blaze README and runtime user guide document the new
endpoint, protection rules, response shape, tombstone flow, and restart
behavior. No changelog entry is added because Blaze has not been released.
To roll back, stop the Blaze daemon and confirm that no sandbox has active
prune scratch (files starting with
.prune.inside a sandbox checkpointdirectory). Revert this commit together with its API, checkpoint-store, and
documentation changes. Any tombstone or owner files left on disk are safe to
delete by hand once the daemon is stopped; the file provider will not
reference them after revert.